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ABSTRACT 

The two phases in the development and tryout of a 
Guidance Counselor Test to be added to the National Teacher 
Examinations Program are discussed* In Phase One, a 150*item written 
test and a 50-item written test based on taped stimulus material %^re 
produced* Each test consisted of five-choice multiple-^cboice 
questions* In Phase Two, the tests %rare administered during May-July 
1972 to a sample of 129 college students who were completing work on 
a Master* 8 Degree in Guidance and Counseling* Five institutions were 
represented in the study* In addition to taking the tests, each 
candidate completed a self-report Questionnaire Data Form* Several 
types of analyses were carried out, including two item analyses (cne 
for the written test and one for the taped test) * An intercorrelation 
matrix showing the correlations between the test and their various 
subscores was also obtained* Ten stepwise regression analyses w^ire 
run, each being based on a continuous variable derived from the 
questionnaire and used as a dependent criterion variable, with the 
written and taped tests used as independent variables* Feedback from 
the candidates about the tests was, in general, positive* Results are 
provided as related to item and test characteristics, 
intercorrelations among the subscores on the written and taped tests, 
and relationships between questionnaire data and test scores* A table 
presents multiple correlations of combinations of taped and written 
subscores with criterion variables* The Questionnaire Data Form with 
scoring specifications and the Test Evaluation Form are provided* 
Tables in an appendix present intercorrelations among the taped test 
and written test, reliabilities of the subscores for the two tests, 
and zero-order correlations of the tests subscores with criterion 
variables* (DB) 
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PHASE ONE; DEVELOPMENT OF THE TEST MATERIALS 



Development of Specifications for the Written Test 

In April, 1971, a Committee of Examiners met at Educational Testing Ser 
vice "^o develop specifications for a Guidance Counselor Test to be added to 
the offerings of the National Teacher Examinations (NTE) Program. The test 
was to be a two-hour test consistent with the other NTE Teaching Area/Field 
Examinations. 

As an initial step, certain guidelines were set in regard to the popula 
tlon for which the test was intended. The test was to focus on guidance 
counselors rather than directors of guidance* It was to be geared to people 
who work in ihe public schools. It was to focus on the secondary school 
level rather than on the elementary school level, although a number of ques- 
tions might have general applicability to both levels. It was assumed to 
be most appropriately taken at ttie end of the master *s degree level of 
training. 

Guidelines were also formulated for the framework to be used In the 
development of the test specifications. The specifications were to be cen- 
tered on the competencies required of the counselor rather than on the 
particular content he or she had studied. These competencies, in turn, were 
to be related to desired student outcomes. In line with this, six major 
categories of counselor competencies and three major categories of student 
outcomes were agreed upon. 

The six major areas of counselor competencies, together with represen- 
tative subcategories, are: 
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I. Interpersonal Communication: Being able to, e.g., give 
accurate feedback of content, reflect feelings accurately, 
vary questions and statements to further communication, 
adapt communication skills to individual and cultural ^/^^-^ . 

differences 

It. Observing and Describing: Being able to, e.g., identify 
relevant behavior to be changed, infer feelings and intent 
from nonverbal cues, precisely describe crucial behaviors 
III. Goal Setting: Being able to, e.g., help client state goals 
as tentative approximations, help client state goals as 
behavior changes likely to produce client satisfaction, 
help client state motives or values for proposed actions 

IV. Systems for Changing Behavior: Being able to, e.g., pre- 
scribe actions that client or others can take to modify 
inappropriate behavior or ^ :it new behavior, such as 
arranging different reinforcement schedules or encouraging 
client base-rate taking; develop educational programs 
directed toward concerns such as anticipating future employ- 
ment work habits or promoting harmonious interracial 
relations 

V. Finding and Providing Information: Being able to, e.g., 
find information client cannot find, interpret information 
that might mislead client, help client avoid confusion by 
falsely considering projections as predictions 
VI. Evaluating Outcomes: Being able to, e.g., judge success on 
basis of client attaining goals, design simple controlled 
experiments, use feedback to modify future procedures 
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The three major categories of student outcomes, together with representative 
subcategories, are: 

1. Learn How to Identify and Anticipate Problems (e.g., group 
conflict resolution, causes of behavior or intent of action) 

2. Learn How to Make Decisions Wisely (e.g., generate alter- 
natives, find information) 

3. Learn How to Take Constructive Action to Effect Change or 
Attain Personal Goals (e.g., overcome fears, make friends) 

Item Writing for the Written Test and Related Validity Concerns 

Prototype items for the written test were produced by the committee mem- 
bers at the April meeting. These were to be incorporated in a manual to be 
sent to item writers. Item writers would include both committee members and 
outside item writers the committee might recommend. The test was to be 150 
items in length. 

A part of the item-writing discussion centered on the use of modes of 
testing other than pencil and paper alone, and the suggestion was made that 
an audio portion be included In the test. Committee members felt that this 
mode would make it possible to obtain a different kind of information about 
the test candidate. He or she might, for example, listen to portions of an 
interview and be asked to identify feelings being expressed or to Identify 
problems from what the client did or did not say. 

Empirical validation was another concern raised in regard to the test. 
Pretesting of the test materials on groups of counselors was considered to be 
a desired procedure. This would also provide feedback about various item 
types that would aid in future test development should a combined written- 
audio test be produced. ^ 



Development o f Specifications for the Taped Test 

As a follow-up of the commiccee's rccommcndaCion chat an audio portion 
of the test be considered, it was decided chat a taped test would be developed 
on an experimental basis and tried out in conjunction with the two-hour, 150- 
item written test that was being developed. In line with this decision, two 
members of the Committee of Examiners met with Test Development personnel at 
£TS in September, 1971, to develop specifications for the taped materials. 

A preliminary matter to be considered in the development of taped 
materials was the division of testing time between the regular written por- 
tion of the test and the taped portion, should a combined package eventually 
materialize in the 4NTE Program. For purposes of reliability, it was decided 
that the taped test ought to consist of at least AO-50 items and the written 
test of about 100 items. Subject to experimentation, division of the two 
hours of testing time for a final test that might be introduced into the 
Program was, Therefore, to be as follows: 

No. Minutes No. Items 

Regular Written Portion 90 100 

Taped Portion 30 40-50 

The time limit and number of items for the experimental taped test was 
to be the same as that above, and the written test was to be a two-hour, 
150-item test as previously indicated. The use of the 150-item written test 
in the tryout was undertaken for several reasons. First, the written test 
was originally conceived as an entity Ir itself and the test development 
process had proceeded on this basis. Secondly, relating the 150'-item test 
as originally conceived by the committee to the experimental taped test would 
provide a better base for determining which categories of the written test 
^ specifications might most fruitfully be measured by written means and which 
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by taped means. Development of a ISO-item written test would also, of course^ 
permit introduction of the Guidance Counselor Test into the NTE Program with- 
out the taped portion if the latter appeared to measure nothing unique* 

Basic ground rules set for the development of the audio portion of the 
test were: 

1. The stimulus materials must be on tape* 

2. The test items were to be of the five-choice, multiple- 
choice type built around the following: 

(a) a single statement by the client 

(b) a single statement by the counselor 

(c) a single client response and counselor response 

(d) multiple client responses and counselor responses 
(brief dialogue) 

(e) a longer excerpt that would lend itself to several 
questions 

3. The test materials were to deal with face-to-face 
interpersonal situations. 

A. The test questions were to be directed to (or refer back 
to) specific topics on the original (written test) 
specifications. 
5. Sets of Items that focused on a given stimulus must be 
limited to 4 or 5 per stimulus, and the stimulus 
excerpt must last no more than 1 1/2 minutes. 
A further guideline in regard to the taped test materials was that appropriate 
balances be maintained among situation variables such as (1) counseling and 
O lulting, (2) clients and counselors, (3) males and females, (4) blacks, 
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and (7) settings to elicit descriptions of what is happening and settings to 
elicit judgments about what is happening. 

In terns of behaviors to be tested, the items were to focus on behaviors 
that require the test taker to: 

1. identify counselor behavior's or techniques 

2. evaluate appropriate counselor behaviors 
3* identify client feelings 

A. describe interaction patterns between counselor and client 

5. select the client response the counselor is most likely to 
elicit 

6. identify group process factors from an excerpt of group inter-* 
action 

Two specific content categories that would provide a focus for stimulus 
materials were also delineated and are indicated below. 





Adolescent 
Developmental Tasks 




Emotional Triggers 
for Adults 


1. 


Self-identity 


1. 


Sex 


2. 


Values 


2. 


Motherhood 


3. 


Sex role 


3. 


Authority 


4. 


Cultural group identification 


4. 


Religior 


5. 


Establishment of Independence 


5. 


Drugs 


6. 


Peer relations 


6. 


Criticism of professional 








competence 



Some of the items in the test woul<l be directed from these content categories 
to specific topics (counselor competencies) in the specifications for the 
written test. Others would be directed only :o the counselor competencies in 
the written test. 
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A few examples may illustrate the use of the specifications* Suppose a 
given excerpt covered a group counseling situation composed of male and 
female students who were discussing the roles of tnen and wot&en in society. 
One of the test questions might be directed toward what the counselor could 
do to further communication among the group. This item could be classified 
as follcA/s: 



Counselor Competency 
Situation Tested in Written 
Variables Test Specifications 



Behavior Tested 
in Taped Test 
Specif icacions 



To promote 
communication 
in groups 



Evaluate 
appropriate 
counselor 
behaviors 



Content Focus 
in Taped Test 
Specifications 

Sex tole 



Counseling 
Males /Females 
Blacks /Whites 
Content 
Group 
Judgment 

Another excerpt might be based on an interaction between a male counselor and 
a female parent in which the parent is expressing hostility toward the coun- 
selor. One of the test questions might be directed toward the feelings the 
client seems to bo expressing. This item could be classified as follows: 



Sltuacion 
Variables 

Consulting 

Female/Male 

White/White 

Feelings 

Individual 

Description 



Counselor Competency 
Tested in Written 
Test Specifications 

Precisely describe 
crucial behaviors 



Behavior Tested 
in Taped Tebt 
Specifications 

Identify 
client 
feelings 



Content Focus 
In Taped Test 
Specifications 
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Item Writing and Developtaent of Taped Stltaulus Materlalg 

As a part of the meeting to develop specifications for the taped test, 
various Item types were discussed. Two item types based on taped examples 
that one of the committee members had brought to the meeting were modified 
to adhere to the flve-*choice, multiple-*cholce format agreed upon* These 
were to serve as partial resource materials for test development. The same 
committee member also offered to make some tapes available to Test Develop- 
ment personnel to be used as possible sources of stimulus materials* 

Several problems emerged in the development of the taped stimulus 
materials* One was trying to get existing taped materials of high enough 
technical quality that they could be reproduced* Related to this was the 
need to consider ethical concerns related to client-counselor confidences and 
to Insure that no materials were used without permission of the parties 
involved. A further problem was that of obtaining sufficient variety in the 
types of counseling situations represented on the available tapes* The 
amount of time required to listen to the available taped materials was rela- 
tively high in relation to the number of excerpts that were isolated as 
potential stimulus materials for item writing* 

Coupled with the above complexities were those Involved in attempting 
to communicate the many facets of the test specifications to others who might 
write test items* It had originally been anticipated that some of the items 
would be written by outside item writers > but because of the problems that 
emerged in exploring the development of the taped materials, most of the 
items for the experimental test were written within ETS* 

The difficulties associated wlt)i obtaining existing taped materials led 
to consideration of what other means might be used to arrive at realistic 
Q counseling situations that could be put on tape* A conference with several 
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people in the area vho ara conceniad with the craining of counaelora waa aet 
up by the Teat Developaent ataff who were working on the teat. Diacuaaion at 
thla neeting led to an offer by one of the ^einona In attendance* who had 
been aaaociated with ETS through Project NOW, to i&ake available hia file of 
loga of actual counaeling aituationa encountered by college atudenta aa part 
of their intern training aa counaelora. By going through thaae loga* Teat 
Development peraonnel were ^le to compile a varied liat of problem altua* 
tlona auf flcienc in number and degree of detail to write the neceaaary 
atlmuluB oaterlala and related teat quea tlona. 

A variety of typea of atimulua materiala and quaatlona were written In 
line with the apeclflcatlona for the taped teat. Theae were grouped into 
five parte* The flrat four parte were baaed on brief taped excerpta and were 
dealgned to aaaeaa the teat taker* a ability to (1) identify client feellnga, 
(2) identify counaelor feelirg8| (3) aelect the probable reaction of a 
client to a particular counaelor reaponaei and (4) evaluate the appropriate- 
neaa of given counaelor reaponaea. The fifth part of the teat waa baaed on 
longer excerpta > including group counaeling aituationa. Theae excerpta were 
dealgned to meaaure a broader range of abilltlea including the ability to 
identify critical laaueai theoretical conatructai counaeling goala* and 
dealred reaponae pattema of counaelora, aa well aa other abilltlea meaaured 
in the flrat four parte of the teat. 

Once the atimulua materiala and queatlona for the teat were preparedt the 
next step waa tu work out the mechanlca of the teatlng altuatlon. Deciaiona 
needed to be made about what parte of the teat materiala were to be put on 
tape, what form the dlrectlona were to take, what the balance of counaelor/ 
client rolea waa to be In terma of aex and ethnic background, what the timing 
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of the vdrloud Item typed uaa to be^ ^nd what the fcinaat of the test book 
should be. A script Incorporating theae decisions could then be prepared 
for use In putting the test materials together. 

Preliminary tapes and test materials for review by the Committee of 
Examiners were prepared when the script was refined. The materials were 
also reviewed by others within ETS who have had training and experience In 
guidance and counseling. EIS staff members served as counselors and clients 
for the preliminary tapes. A high school drama class was to be used la the 
final taplngf In accordance with recommendations by comnlttee members who 
developed the specifications for the taped test. 

Review of the preliminary test materials was positive* in general* The 
excerpts were considered to be characteristic of realibtic counseling situa- 
tions and to represent a balance of counseling situations. The item types 
In the taped test were considered to measure abilities not covered in the 
written test. The possible responses to items were considered to cover an 
array of plausible distracters. 

Negative comments by reviewers related primarily to the more technical 
aspects of the test materials. For example » it was suggested that more time 
was needed for some of the questions and that minor modifications might be 
helpful in regard tc the test directions. Other comments were made concern- 
ing the quality of the tapes in that it was not always possible to hear 
clearly all of the voices. 

In view of the generally positive response to the taped test, a final 

script was prepared that took into account desired changes in the timing of 

some of the excerpts, the wording of some of the directions, and other 

suggestions that had been made by those who reviewed the test. Final taping 
O 

ERJChe experimental teat was subsequently undertaken at a professional 
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recording studio* Representatives from the Test Development and Program 
Divisions of ETS served as counselors. The drama class of a local high school 
served primarily as the clients, with assistance from two other young people 
associated with ETS. The directions for the test were recorded by one of the 
NTE Program Directors. 
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PHASE TWO: TRYOUT OF THE TEST M/\TERIALS» /\NALYSIS OF 
DATA, AND FINDINGS OF THE STUDY 

Pcscrlptlon of the Sample and Procedures for the Tryout and Analysis of Data 

The tests were administered during May-July, 1972, to a sample of 129 
college students (f « 80, m " 49) who were completing work on a Master's 
Degree in Guidance and Counseling. About one-third of the students had 
undergraduate majors in Education and about one-fifth had undergraduate 
majors in Psychology. The remainder had been primarily Social Science and 
Humanities majors. 

Five institutions were represented in the study, including a New England 
college, two California universities, and two Southern universities. Admin- 
istrative arrangements for the tryout of the materials were made through the 
assistance of members of the Committee of Examiners and other ETS contacts 
in the field. Score reports were to be provided to institutions and students 
wlio took part in the study. 

The tests were administered under conditions similar to a regular 
administration in that a supervisor's manual outlining testing procedures 
had been prepared. Each candidate took the standard written test consisting 
of 150 five-choice, multiple-choice questions and a 50-item test (also five- 
choice, multiple-choice) based on taped stimulus material. Because of the 
amount of testing time required, however, the written test; was taken in one 
sitting and the taped test in another sitting. 

In addition to taking both tests, each candidate completed a self-report 
Questionnaire Data Form, which the Test Development staff had prepared, 
^ vering such items as (1) number of graduate hours in guidance and counsel- 
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Ing, (2) expected level of training sought, (3) number of graduate hours 
devoted to intem/practicum experiences, (A) grade-^point average in guidance 
and counseling courses, (5) grade-point average in intern/practicutn courses, 
(6) years of teaching experience, (7) undergraduate major, (8) part-time 
counseling experiences as a teacher, (9) out-of-school counseling experiences 
(10) experience directing extracurricular activities, (11) focus of counsel- 
ing interest, (12) desired work setting, (13) practical/ theoretical balance 
of training, (14) theoretical emphasis of counseling training, and (15) ethni 
origin. A copy of the questionnaire may be found in Appendix A, page 29. 
Feedback from candidates was also requested through a Test Evaluation Form 
prepared by the Test Development staff. This form is also found in Appendix 
A, page 33. 

Several types of analysis were to be carried out as a part of the study. 
First, two item analyses were to be requested — one for tha written test and 
one for the taped test. These analyses would provide information concerning 
the discriminating power and the level of difficulty of the test questions 
and their concomitant statistical usefulness as prototype items for the 
operational tests. 

An intercorrelation matrix showing the correlations between the tests 

and their various subscores was also to be obtained. The correlation data 

would be helpful in assessing the uniqueness of subsets of test questions and 

provide information useful for future test development. Subscore information 

was to be for internal use only and was to be based on the following break^ 
it 

down of scores: 



Numbers in parentheses indicate number of items per subscore. 
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Written Tesr 



Taped Test 



1* Interpersonal 

communication (32) 



2. Observing and 
describing (17) 

3. Goal setting (26) 



4, Systems for behavior 
change (30) 

5. Finding and providing 
information (23) 



1. Identifying or describing 
counselor goals, responses, 
techniques (10) 

2. Evaluating counselor 
behaviors , responses (15) 

3. Identifying client feelings, 
problems, critical issues (17) 

4. Selecting client response 
counselor is most likely to 
elicit (8) 
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6. Evaluating outcomes (22) 
It should be noted that for purposes of analysis, the subscores on the taped 
test were compressed into four categories of behaviors because of the small 
numbers of items under some behaviors on the specifications for the taped 
test. 

Regression analysis was a further means of exploring the data collected. 
Relating test score information to criterion variables covered by the ques- 
tionnaire would provide information concerning criteria assumed to be relevant 
to performance as a guidance counselor and the related construct validity of 
the tests. 

Ten stepwise regression analyses were run, each of which was based on a 
continuous variable that was derived from the questionnaire and used as a 
dependent criterion variable, with the written test (six subtests) and taped 
test (four subtests) used as independent variables. Sex was arbitrarily 
treated as a continuous (criterion) variable in the regression analyses. A 
verbal aptitude measure was used as a concomitant variable (covariate) in 
the regression analyses in order to control for institutional differences in 
ability. 
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Five categorical variables that were also derived from the questionnaire 
(Undergraduate Major, Race, Vocational Plans, Type of Training, and Desired 
Type of Counseling Setting) were studied separately by examining the distri- 
butions of the total tape and written scores for each of the five variables* 

Feedback from Candidates About the Tests 

Comments from candidates reported on the Test Evaluation Form (see 
Appendix A, page 33) are too numerous to report In detail* However, some 
highlights based on those who returned the form (N « 115) are reported below: 

* About three-fourths of the candidates felt the taped test 
measured different kinds of abilities than those measured 
by the written test. Unique characteristics mentioned 
were "ability to respond to tone of voice, feeling," 
"more measurement of those factors crucial to counseling 
like empathy and comprehension," "listening. ..perhaps 

the most Important skill In counseling," "ability to 
remember exactly what was said," "[covered] more practical 
aspects of counseling." 

* Candidates were about evenly divided as to whether they 
felt they had sufficient time to answer the questions. 
Negative comments Included "did not get sufficient time 
to analyze responses," "yes In first [parts], no In 
longer excerpts," "not enough time to read and consider 
the best answer," "on these type questions I really like 
to think In detail," "the time gave little time for 
evaluation of choice selections." 
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• Practically all of the candidates felt the directions were 
clear. In some cases, it was suggested that repeating the 
directions would be helpful. 

• About three-fourths of 'he candidates felt the technical 
quality of the tape was satisfactory. Some comments were 
"good acting," "excellent right down to accents and intona- 
tion," "most of [the excerpts] were clearly heard, but a few 
were not clear," "sometimes I felt the people speaking spoke 
too quickly." Conditions of the testing situation were 
also mentioned — e.g., too hot, too many extraneous sounds, 
inappropriate sound control. 

• The types of questions in the taped test that candidates 
appeared to consider most effective in measuring counselor 
ability were those related to the nature of or appropriate- 
ness of the counselor's response to a client. 

In general, feedback from candidates was positive in nature, as is indi- 
cated above. Comments suggested that the taped test was essentially sound 
and useful for its intended purposes. 

Two particular concerns in regard to the taped test might be noted. 
First of all, being certain that the listening conditions are good is an 
important consideration in administering the test, as indicated by candidates* 
responses. The timing of the various excerpts and questions is a more com- 
plex concern in that part of the measure of an effective counselor is his or her 
ability "to listen attentively," "to comprehend or perceive quickly," and the 
like. More data probably are needed as an information base in making deci- 
sions related to the timing of taped materials of this kind. 
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Item and Test Characteristics (N « 115) 

Number Raw Raw ReW Score Mean 

of Score Score Standard Item Mean Item Relia*- 
Items Range Mean Deviation A* r biserial bility 

Written Test 150 5-129 89.27** 25.46 11.04 .39 .94 

Taped Test 50 8-45 31.65** 7.07 11.50 .43 .82 

As may be noted from the mean raw score and mean A» the tests were some- 
what easy for this population — the written test slightly more so than the 
taped test. If the tests had been of middle difficulty, the raw score means 
would have been 75 and 30 on the written and taped tests, respectively, and 
the mean delta for each test would have been 12.0. The raw score ranges and 
score standard deviations, relative to the numbers of items, are in accor- 
dance with those of most of the other Area Examinations in the NTE Program. 

The mean r biserials indicate that the majority of items in the tests 
had the power to discriminate between candidates who scored high on the test 
and those who scored low on the test, using total score on each test as the 
criterion. The aim for tests related to education is for a mean lower limit 
r biserial of .30, and .40 is generally about the mean upper limit obtained. 

Reliability coefficients were computed by a modification of Kuder- 
Richardson Formula 20. The estimated reliabilities for the two tests also 
reflect the tests* discriminating power, with .90 being the desired lower 
limit for a 150-item test related to education. Had the written test been 
100 items, the number of items projected for the written portion of the 



A of 13.0 » 50% of population responding to the item answered it correctly. 
As lower than 13.0 indicate easier items — e.g., A12 « 60% answered item 
correctly. 

Scores on the written test reflect application of the standard penalty for 
guessing formula. Scores on the taped test are based on rights only. 
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operational final form, the estimated reliability would have been .91 (accord- 
ing to application of the Spearmar-Brown formula). Since reliability Is 
related to length of test, the reliability of .82 for the taped test of SO 
Items can be considered quite acceptable. Assuming that a final form test 
would be prepared that consisted of 100 Items In the written portion and 50 
Items In the taped portion, the estimated reliability of the total test would 
be expected to meet the desired criterion of .90 or higher. 

All of the data above should be considered only suggestive In the light 
of the limited sample of 115 cases. The lower-limit sample size for item 
analysis of tests of this kind is usually about 300 cases. 

Intercorrelations Among the Subscores on the Written and Taped Tests 

The correlation between scores on the total written test and scores on 
the total taped test was .81. This is a high correlation in relation to the 
estimated test reliabilities of .94 and .82, and it suggests that those who do 
well on the written test also do well on the taped test. 

The intercorrelations among the subscores are less easily Interpreted 
because of the variations in the number of items in each sub score and the 
small numbers of items upon which some subscores are based. The range is 
from 8 items for Subscore 4 of the taped test (see page 14) to 32 items for 
Subscore 1 of the written test. However, a few highlights are summarized 
below: 

• Correlations between Subscores 1, 2, 3, and 4 of the taped 
test and the total written test were .62, .73, .64, and .57, 



erJc 



A complete table of intercorrelations, as well as the reliabilities of the 
various subscores, may be found in Appendix B, page 37. 
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respectively, which suggests that Subscore 2 (Evaluating 
counselor behaviors, responses) is more highly related to 
candidates* scores on the written test than any of the 
other subscores based on the tape. However, had the 
various subscores and the written test been perfectly 
reliable measures, the Intercorrelations between Subscores 
1, 2, 3, and A of the taped test and the total written 
test would have been .91, .96, .83, and .99, respectively, 
as revealed by correction of the intercorrelations for 
attenuation (or the effects of errors of measurement). 
With such correction, Subscore 4 (Selecting client 
response counselor is most likely to elicit) is corre- 
lated more highly with the total written test than is 
Subscore 2 (Evaluating counselor behaviors, responses). 
Subscores 1, 2» 3, 4, 5, and 6 of the written test corre- 
lated .74, .66, .74, .73, .63, and .65, respectively, 
with the total taped test, which suggests that Subscores 
1 (Interpersonal communication), 3 (Goal setting), and 
4 (Systems for behavior change) of the written test show 
the highest correlations with the total taped test. 
Correction of the intercorrelations for errors of 
measurement reveals that had the subscores on the written 
test and the total taped test been perfectly reliable, 
the intercorrelations between Subscores 1, 2, 3, 4, 5, 
and 6 of the written test and the total taped test would 
have been .93, .86, .92, .89, .85, and .88, respectively. 
Written Test Subscores 1 (Interpersonal communication) 
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and 3 (Goal setting) thus continue to show the highest 
correlation with the total taped test. However^ Subscore 
6 (Evaluating outcomes) is almost as highly correlated 
with the total tiped test as is Subscore A (Systems for 
behavior change). 

Relationships Between Questionnaire Data and Test Scores 

Study of the five categorical variables derived from the Questionnaire 
Data Form revealed that two of the variables, Undergraduate Major and Racey 
produced significant differences in terms of mean test score performance. 
There was a tendency for candidates with undergraduate majors in Humanities, 
Psychology 9 and the Social Sciences to do somewhat better on both tests 
than candidates with undergraduate majors in Education, Science-Mathematics, 
or Languages. The 100 white candidates who took the tests tended to score 
higher on both tests than did the 15 black candidates* 

Regression analyses based on 105 cases from four institutions (incom- 
plete information was available for the fifth institution) revealed that 
parts of the taped test in conjunction with parts of the written test are 
more related to the criterion variables than either the taped test or the 
written test alone. (The dependent criterion variables were not necessarily 
' experimentally Independent of each other.) Appendix B» page 38» shows the 
.ntercorrelatlona among the covariate, the predictors » and the criterion 
variables prior to the regression analyses. 

The table below lists the ten dependent criterion variables and shows 
the multiple correlations observed using the best combination of written 
and tape subscores with a verbal aptitude measure partialed out. Criteria 
be met for continuing or stopping the stepwise regression analyses, based 

ERIC 
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on proportion of the variance accounted for by the measurest were •0125 or 
more to enter a s^easure and •00975 or le'^s *o drop a measure^ The scoring 
scheme for each of the variables In the questionnaire may be found In 
Appendix At page 31^ 
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Table 1 Multiple Correlations of Combinations of Taped and 
Written Subscores with Criterion Variables 

Multiple 
Correlations 

Best Single Predictor of Criterion, (with verbal 

Dependent Criterion Together with Added Predictors measure par*- 

Variable WS » Written Subscorc; TS » Taped Subscore tialed out) 



Graduate Hours in WS 5 (Finding and providing information) .27 

Guidance and Coun- TS 2 (Evaluating counselor behaviors, .31 
seling responses) 

WS 2 (Observing SO^^^escribing) •^^^ 

Expected" Level of WS"5^EvaIuating''outcomc8T 7l8 

Training Sought 

Graduate"Hour8"in i3s"2"T6b8erving and descriSIngT* ,19" 

Intem/Practicum TS A (Selecting client response counselor .25 
Courses is most likely to elicit) 

TS 1 (Identifying or describing counselor .28* 

Grade"Point''Average TS''2"'J Evaluating counselor "behaviors, " ~ Tl? 

Graduate Courses in responses) 

Guidance/Counseling WS 2 (Observing and describing) .23 

WS 5 (Finding and providing information) .27 
TS A (Selecting client response counselor .30 
is^most_likcl^^to^elicit) 

Grade"PoInt"Average fs"3"n3cnti?ying*client^ccnngs, problems, .25 

in Intern/ critical issues) 

Practicum Courses WS 2 (Observing and describing) .27 

WS 5 (Finding and providing information) .31 
WS 3 (Goal setting) .33 
TS A (Selecting client response counselor .37* 

is^most^likel^to^llcit) 

Years^of "Teaching TS"3''ndentifying client Feelings, problec^T 73?** 

Experience £Ei!i£lESi.iSSH£2) ^ 

Part-time''Coun- i5s"l"Tlnterper8onaI communicationy 715* 
seling Experiences WS A (Systeass for behavior change) .26 
as a Teacher WS 5 (Finding and providing information) .30 

TS A (Selecting client response counselor .33* 

is^most^likely^to elicit) 

Directioa''of "School WS"5"TFin3ing"an3"provi3ing"inforM " 725* 

Extracurricular TS 1 (Identifying or describing counselor .31 
Activities/Counseling goals, responses, techniques) 

with Youth Groups WS A (Systems for behavior change) .35** 
in^Communit^ ^ 

Theoretical versus WS*i"(interper8onal~ommunica ""•31 
Practical Training WS 6 (Evaluating outcomes) .3A** 
Emphasis^ « « 

Sex TS"3"ndentifying"clicnt"Feeli^^ "".l5 

critical issues) 
WS 1 (Interpersonal communication) .29 

i?^«2.i2^2£*^lI^SB S^^-^^Si'^i'^^DSl.-.^ • 



Significant at .01 level 
Significant at .05 level 
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A nutabcr of observations of possible interest may be made on the basis of 
the table. Aaong these are: 

• Partialing out a verba^ neasure (or trying to eliminate the 
effect of verbal ability upon test performance) revealed 
that the taped test makes more unique contributions to the 
total Guidance Counselor Test than is suggested by the 
relatively high intercorrelation between the written test 
and the taped test indicated on page 18. The taped test 
contributes to the prediction of 8 of the 10 criterion 
variables listed* 

• Performance on Subscore 3 of the taped test (Identifying 
client feelings* problems « critical Issues) was the best 
single predictor of Grade Point Average (GPA) In Intern/ 
Practicum courses* assuming the self-report information 
about GPA Is accurate. Higher grades tended to be 
associated with higher scores on Subscore 3« 

• Performance on Subscore 3 of the taped test (Identifying 
client feelings* problems* critical Issues) was also 
significantly related to Years of Teaching Experience. 
There was a tendency for those who achieved higher scores 
on Subscore 3 to have fewer years of teaching experience. 
However* since about 40 per cent of the sample had no 
teaching experience* it Is difficult to assess the pre- 
cise nature of the relationship between the two measures. 
None of the subscores on the written test had signifi- 
cant predictive value In regard to Years of Teaching 
Experience. 

ERLC 
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There was a tendency for women to achieve higher scores than 
men on Subscore 3 of the taped test (Identifying client feel- 
ings, problems t critical Issues). Subscores 1 and 2 of the 
written test (Interpersonal communication and Observing and 
describing) also had predictive value in regard to sex of the 
test taker. 

Subscore 1 of the written test (Interpersonal communication) 
is the best single predictor of Theoretical versus Practical 
Emphasis in counselor training. Those who scored high on 
Subscore 1 of the written test tended to indicate that their 
training had been more theoretical than practical. Sub- 
score 6 of the written test (Evaluating outcomes) also contri- 
butes to the prediction of Theoretical versus Practical 
Training Emphasis. 

The best single predictor of Part-time Counseling Experiences 
as a Teacher was Subscore 1 of the written test (Inter- 
personal communication)* Those who had had such experience 
tended to receive lower scores on this subscore « Subscores 
4 and 5 of the written test (Systems for behavior change and 
Finding and providing information) and Subscore 4 of the 
taped test (Selecting client response counselor is most 
likely to elicit) also contribute to the prediction of this 
variable. The specific nature of the relationships between 
these measures is difficult to assess because » as previously 
indicated » such a sizable proportion of the sample had had 
no teaching experience. 
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Those who Indicated chat chey had directed school extra- 
curricular activities or had had experience working with 
youth groups in the community tended to receive higher 
scores on Subscore 5 of the written test (Finding and 
providing information). Subscore 1 of the taped test 
(Identifying or describing counselor goals, responses, 
techniques) and Subscore 4 of the written test (Systems 
for behavior change) also contribute to the prediction 
of these variables. Again, however, precise interpre- 
tation of the data is compounded by the need to take 
into account the sizable proportion of the sample 
which had had no teaching experience and the fact 
that two variables were combined in the regression 
analysis. 

Subscore 2 of the written test (Observing and describ- 
ing) was the best single predictor of number of graduate 
hours in Intern/practlcum courses. Those who received 
higher scores on this subscore tended to have more 
graduate hours in such courses. Taped Subscores 1 and 
4 (Identifying or describing counselor goals, responses, 
techniques and Selecting client response counselor is 
most likely to elicit) also had predictive value in 
regard to this variable. 

Subscore 5 of the written test (Finding and providing 
information) was the best single predictor of tiumber of 
graduate hours in guidance and counseling. Higher 
scores were associated with higher number of graduate 
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hours. Subscore 2 of the written test (Observing and describ- 
ing) and Subscore 2 of the taped test (Evaluating counselor 
behaviors, responses) also contribute to the prediction of 
number of graduate hours in guidance and counseling. 
In interpreting the data above, one should keep in mind the fact that 
intercorrelations among the dependent criterion variables are ignored. It 
is quite likely that many of them, if not all, are intercorrelated and that 
measurement of the same thing is being repeated from variable to variable. 
The fact that the questionnaire data are based on self-report information is 
another limitation. 

Despite the number of limitations associated with a pilot study of the 
sort undertaken in this investigation, there appears to be evidence that the 
taped portion of the Guidance Counselor Test provides information that is 
unique to some of that provided by the written portion of the test and that 
this information is associated with important types of educational measures. 
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ID 

(from card given to you) 

QUESTIONNAIRE DATA FORM 
(Guidance Counselor Tests) 

The information below is being collected for research purposes. It will 
not affect your score. Your cooperation in completing the questionnaire will 
be very much appreciated. 

Name 

Last First Middle Init. 

Date of Birth Sex 

Month Date Year 

Home AddresF 



Institution 



DIRECTIONS : Please answer the following questions by circling the letter of the 
response that best applies to you. Mark only one response for each question. 

1. a) If your credits are in semester hours, how many semester hours of gradu- 

ate work in guidance and counseling will you have completed by June, 1972? 

(A) 10 or less (B) 11-15 (C) 16-20 (D) 21-25 (E) 26-30 (F) 31+ 

b) If your credits are in quarter hours, how many quarter hours of graduate 
work in guidance and counseling will you have completed by June, 1972? 

(A) 10 or less (B) 11-15 (C) 16-20 (D) 21-25 (E) 26-30 (F) 31+ 

2. At which of the following poi..ts do you expect to terminate your training 
in guidance and counseling? 

(A) Masters Degree (B) Masters Degree + 30 hours (G) Doctorate 

3. a) If your credits are in semester hours, how many semester hours of your 

graduate work will have been devoted to intern/practicum experience by 
June, 1972? 

(A) 3 or less (B) 4-7 (C) 8-12 (D) 13+ 

b) If your credits are in quarter hours, how many quarter hours of your 
graduate work will have been devoted to intern/practicum experience by 
June, 1972? 

(A) 3 or less (B) 4-7 (C) 8-12 (D) 13+ 

4. Using the following scale, indicate your grade-point average in your gradu- 
ate courses in guidance and counseling to date. (A=5;B«4;C=3) 

ERJC (A) 5-0 (B) 4.7-4.9 (C) 4.4-4.6 (D) 4.1-4.3 (E) 3.8-4.0 (F) 3.7 or below 
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5. Using the same scale, indicate your grade-point average in your intern/ 
practicum courses. 

(A) 5.0 (B) A. 7-4. 9 (C) 4.4-4.6 (D) 4.1-4.3 (E) 3.8-4.0 (F) 3.7 or below 

6. How many years of teaching experience have you had? 

(A) None (B) 1 (C) 2 (D) 3 (E) 4 (F) 5 (G) 6 (H) 7+ 

7. In which of the following categories would you classify your undergraduate 
major? 

(A) Education (B) Psychology (C) Science-Mathematics (D) Social Studies 
(E) Humanities (F) Languages (G) Other 

8. Did your work as a teacher, if any, include any part-time counseling of 
students? 

(A) Yes (B) No 

9. Have you had any counseling experiences other than those required as part 
of your graduate training or as a teacher (e.g., working with youth groups 
in the community)? 

(A) Yes (B) No 

10. Did your work as a teacher, if any, include directing any extracurricular 
activities? 

(A) Yes (B) No 

11. In which of the following areas of counseling are you most interested? 
(A) Educational planning (B) Vocational planning 

(C) Dealing with personal problems 

12. In which of the following settings do you plan to work? 

(A) Elementary school (B) Secondary school (C) Junior or community college 
(D) Four-year college (E) Business-industry (F) Private agency 
(G) Community agency, rehabilitation, employment services, YMCA, YWCA, etc. 

13. Which of the following best describes the practical/theoretical balance 
of your training? 

(A) More theoretical than practical (B) More practical than theoretical 
(C) Fairly evenly balanced between theory and practice 

14. How would you describe the primary theoretical emphasis of your training? 
(A) Rogerian (B) Behavioral (C) Directive (D) Psychoanalytic 

(E) Rational (F) I consider my training to be eclectic. 
.15. Which of the following best describes your ethnic origin? 

(A) American Indian (B) Black, Afro-American (C) Mexican-American 
(D) Oriental (E) Puerto Rican (F) Spanish-American 
(G) White (H) Other 
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Scoring Specifications for the Questionnaire Data Form 



Item II 


A 


B 


C 


D 


E 


F 


■ G 


H 


Omit 


1 (a) 


12 


20 


27 


34 


42 


50 






1 . 
























25 


1 (b) 


08 


13 


18 


23 


28 


33 










2 


1 


2 


3 












2 


3 (a) 


02 


08 


15 


22 


































9 


3 (b) 


02 


06 


10 


15 














4 


5 


4 


3 


2 


1 


0 






2 


5 


5 


4 


3 


2 


1 


0 






2 


6 


0 


1 


2 


3 


4 


5 


6 


8 


2 


7* 


1 


2 


3 


4 


5 


6 


7 




0 


8 


2 


4 














3 


9** 


2 


4 














3 


10** 


2 


4 














3 


11* 


1 


2 


3 












0 


12* 


1 


2 


3 


4 


5 


6 


7 




0 


13 


3 


1 


2 












2 


14* 


1 


2 


3 


4 


5 


6 






0 


15* 


1 


2 


3 


4 


5 


6 


7 


8 


0 



*Categorical variables 

**A single score was obtained for questions 9 and 10 in combination. It 
was computed by adding the weights assigned to the options that the 
subject selected and subtracting the sum from 9. 
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ID No« (Optional) 

TEST EVALUATION FORM 
(Guidance Counselor Tests) 

To the Candidate ; It will be of help In planning future test development 
activity if you will record on this form your reactions to the tests you 
have taken. Your assistance is very much appreciated. 



1. Did you feel that the taped part of the test measured different kinds of 
abilities than those measured by the written test? Yes No 

Comments: 



2. Was there sufficient time for you to answer the questions in the taped 
test? Yes No 



Comments: 



3, Were the directions for the taped test clear? Yes No 

Comments: 



4. Was the technical quality of the tapp such that the voices were clearly 
heard? Yes No 



Comments : 



ERLC 
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5. What kinds of questions In the taped test did you think were most effec- 
tive in measuring counselor ability? 



6. Was there sufficient time for you to answer the questions on the written 
(longer) test? ^Yes No 

Comments : 



7. What were your reactions to the written (longer) test as a whole? 



Additional Comments About Either Test: 
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Table Bl 

IncercorrelaClons Among the Taped Test and the Written Test 





Total 


TS 


TS 


TS 


TS 


Total 


WS 


WS 


WS 




Tape 


1 


2 


3 


4 


Written 


1 


2 


3 


TSl 


.75 


















IS 2 


.85 


.50 
















TS3 


.86 


.54 


.59 














TSA 


.69 


.44 


.47 


.47 












Total 


. ox 




• i J 




. 57 










Written 












WSl 


.74 


.54 


.70 


.59 


.50 


.88 








WS2 


.66 


.48 


.59 


.51 


.50 


.81 


.70 






WS3 


.74 


.58 


.64 


.59 


.52 


.90 


.79 


.67 




WSA 


.73 


.54 


.67 


.57 


.52 


.91 


.73 


.72 


.78 


WS^ 
WS6\ 


.63 


.50 


.58 


.48 


.44 


.84 


.67 


.60 


.70 


.65 


.53 


.55 


.54 


.43 


.79 


.57 


.55 


.66 



WS WS WS 
4 5 6 



71 



Table B2 



Reliabilities of the Subscores for the Written and Taped Tests 

(N > Num1)er of Items) 



TSl 


TS2 


TS3 


TS4 


WSl 


WS2 


WS3 


WS4 


wss 


WS6 


N-10 


N-15 


N-17 


N-8 


N-32 


N-17 


N-26 


N>30 


N-23 


N-22 


.49 


.61 


.63 


.35 


.78 


.70 


.77 


.83 


.73 


.67 
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